Final Report for ZSY Playing

Introduction We aimed to train an agent through Reinforcement Learning to beat a human at the card game ZSY. The inputs are the game states at each particular turn and the outputs are the optimal move given the input game state. Below are the abbreviated game rules: 1. A game consists of a number of rounds and each round has a number of turns. Both players are dealt 18 random cards (their ‘hand’s) at the start. On the first round, one player is randomly selected to have the first turn. That player plays a pattern which can be one of the following: ‘single’ (a single card, e.g. 5), ‘double’ (two cards of the same rank, e.g. 99), ‘triple’ (three cards of the same rank, e.g. 777), ‘bomb’ (four cards of the same rank, e.g. 4444), or a ‘chain’ (consecutively ranked cards in which each ‘link’ is at least a double, e.g. 6677788 but not 66788 because there are fewer than 2 7s or 6688 because the ranks are not consecutive). 2. At each following turn, a player can play cards that match the pattern and have higher rank (e.g. 555 can be followed by 777 but not 88 or 444; 55666 can be followed by 77888 but not 77788 or 33444), play a bomb, or pass. Players alternate turns until one player passes, and the round terminates. The last player to play some cards in the round begins the next round and sets a new pattern. The game terminates when one player runs out of cards and wins the game. To model this game, we built a game class in python that deals cards, keeps track of the running card counts, and sends game state data to the appropriate agent class that return a move according to that agent’s policy. This was a joint project with CS 221, but in that class we approached a different version of the game in which both players could see the other’s cards. In that case, the game is theoretically deterministic from the perspective of the agents, and we tried to design a minimax agent rather than a learning agent. The only shared portion was the game class we wrote, modified to send full game state data (i.e. hands of both the agent and its opponent) to the agents for 221 rather than partial game state data (i.e. the hand of the agent and the number cards the opponent has left) for 229. We tried two RL algorithms: Q-learning and TD-learning. In the first, we tried to find the values for states coupled with actions (the move taken), and in the second we tried to find the values for just the states.